Evaluation of MT Systems by TOEFL

Recently, we did some comparative evaluation of the commercial English-to-Japanese MT systems by using the TOEFL, Test of English as a Foreign Language. The passages in the Reading Comprehension part of the Vocabulary and Reading Comprehension section were extracted from a TOEFL guide book [1], and were translated into Japanese by using the MT systems. Then the translated tests were taken by the examinees who are native speakers of Japanese as if they were Japanese comprehension tests. The scores of the tests would reflect the systems' abilities to convey the semantic contents of the input passages to the examinees. This paper describes what we have learned from the experiments.